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Amendment to the Specification 



Please replace the paragraph on page 1, lines 3-7, with the following amended 
paragraph: 



0 



The present invention relates to image processing, in particular to methods and 
apparatus for registration and merging of a plurality of overlapping two-dimensional 
(2D) images of three-dimensional (3D) scenes, especially in cases where the 2D 
images are related to the 3D scenes by a projective (or camera) transformation. The 
invention also r e lat e d relates to apparatus for performing the disclosed image 
processing. 



Please replace the paragraph on page 2, lines 13-15, with the following amended 
paragraph: 

The objects of the present invention are to provide apparatus and methods 
which simply and stably determine an accurate projective transformation relating two 
imag es from pairs of correspondi ng points identified in each image of the pair. 



Please replace the paragraph on page 2, lines 16-21, with the following amended 
paragraph: 

Achieving these objects by the present invention depends on the discovery that a 
projective transformation can be better determined from a plurality of pairs of 
corresponding points in both images when the numerical ranges of the coordinates of 
these corresponding points is are minimized. With minimum numerical ranges of the 
coordinates, errors arising from terms non-linear terms in the coordinates are reduced 
in comparison with other methods which are ignorant of this discovery. 



Please replace the paragraph on page 2, lines 22-28, with the following amended 
paragraph: 

The present invention minimizes these coordinate ranges prior to determining 
the projective transformation relating a pair of images. In one alternative, an original 
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coordinate system is chosen in each image in advance to minimize these coordinate 
ranges. This choice can be made, for example, by finding a coordinate origin for which 
the sum of the radius vectors to the feature points is a minimum. Such a coordinate 
origin can be found by a search technique. In this alternative, the projective 
transformation is determined directly in the origjnal coordinate system. 

Please replace the paragraph on page 3, lines 17-29, with the following amended 
paragraph: 

In detail, these objects are achieved by the following embodiments of this 
invention. In a first embodiment, the present invention includes a method for merging a 
pair of overlapping two-dimensional (2D) images, said images being projections of a 
single three-dimensional (3D) scene, said method comprising: selecting at least four 
feature points in the 3D scene, finding the 2D coord i nat e coordinates of the points in 
both images corresponding to the selected feature points, the 2D coordinates being 
found with respect to original coordinate systems in the two images, translating the 
original coordinate systems of the two images in order to substantially minimize the 
average coordinate ranges of the 2D coordinates found, determining the parameters of 
a substantially optimal projective transformation relating the corresponding translated 
coordinates in the two i mag e images , determining the parameters of the projective 
transformation for application in the untrans l at e d non-translated coordinate systems of 
the two images, and merging the two images by transforming one image according to 
the projective transformation and combining the transformed image with the other 
image, . 



Please replace the paragraph beginning on page 3, Iine30 through page 4, line 2, 
with the following amended paragraph: 

In a second embodiment, the invention includes an apparatus for merging a pair 
of overlapping two-dimensional (2D) images, said images being projections of a single 
three-dimensional (3D) scene, said apparatus comprising: means for obtaining a pair of 
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i Jt 2D images, a processor responsive to the means for obtaining images and configured 
to perform the methods of the first embodiment, and a display for viewing the pair of 
images merged by the processor. 



Ai 



Please replace the paragraph on page 4, lines 3-9, with the following amended 
paragraph: 

In a third embodiment, the invention includes an x-ray apparatus for merging a 
pair of overlapping two-dimensional (2D) images, said images being projections of a 
single three-dimensional (3D) scene, said apparatus comprising: an x-ray source for 
projecting a beam of x-rays through an object to be examined, an x-ray detector for 
obtaining digital x-ray images which are projections of the object, a processor 
responsive to pairs of overlapping x-ray images obtained by the x-ray detector and 
configured to perform the methods of the first embodiment, and a display for viewing 
the pair of images mer g ed by the processor. 



Please replace the paragraph on page 4, lines 10-12, with the following amended 

paragraph: 

In a fourth embodiment, the invention includes a computer readable medium 
comprising encoded program instructions for causing a processor to perform the 
methods of the first embodiment 

Please replace the paragraph on page 5, lines 9-15, with the following amended 
paragraph: 

Digital 2D images of 3D scenes can be obtained for input to the apparatus of this 
invention for processing by the methods of the present invention by any means known 
in the art. One exemplary means is to simply scan standard photographs with a digital 
scanner. Fig. 1 illustrates another exemplary means, digital PC camera 21 which 
includes lens system 23 for optical imaging and CCD array 22 for conversion of an 
optical image to digital signals for input to PC 28 27. As illustrated, an image of a 3D 
scene including arrow 31 is proje cted onto CCD array 22 through lens syste m 2 3. 
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Please replace the paragraph on page 5, lines 16-23, with the following amended 
paragraph: - — — . 



Fig. 2 illustrates use of digital camera 32, perhaps of greater capability than PC 
camera 21 , which is mounted on tripod 33 for rotation about axes 34 and 35 in order to 
pan across extended scene 37. Since camera 32 can form images of only a limited 
part of the 3D scene at one time, for example, of objects in cone 36 which is projected 
onto a digital pickup in camera 32, forming an image of entire extended scene 37 
requires than that multiple individual images be merged into a composite image by the 
image processing apparatus according to the present invention. Digital camera 32 can 
be responsive to selected bands of electromagnetic radiation, for example to infrared or 
to visible light. 



Please replace the paragraph beginning on page 5, Iine33 through page 6, line 5, 
Mthjh<^foHowln 

For example, patient 12 can be longitudinally displaced along direction 11 on 
patient table 8 by motor means 9. The x-ray source and the x-ray detector, mounted on 
C-arm 3 that is in turn mounted by collar 4 on support 5, are capable of coordinated 
rotation about tow two perpendicular horizontal axes 12' and 12". Finally, support 5 can 
be longitudinally translated along rails 7 or rotated about vertical axis 6. The present 
invention is also applicable to x-ray apparatus with other means for jointly moving the x- 
rav source and the x-ra y detector for rotation or translation. 



Please replace the paragraph on page 6, lines 6-18, with the following amended 
paragraph: 



To obtain a composite image of an extended region of the patient, for example, 
of the patient's legs, a plurality of individual images formed during panning of the x-ray 
apparatus about one or more of these degrees of freedom must be merged. Fig. 4 
illustrates in a diagrammatic fashion the formation of limited images and the merging of 
consecutive and overlapping images into an assembled extended image. A patient's 
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leg 16 is shown on patient table 8. Vertical support 5 is moved along rails 7, so that the 
x-ray source is moved in the direction of arrow 14. As the x-ray source is moved, x-ray 
jj beam 15 is intermittently directed at the patient's leg. Together with the x-ray source^ 
the x-ray detector is also moved so as to face the x-ray source when the patient is 

v J* 

^ irradiated. Whenever the patient's leg is irradiated^ a limited x-ray image is formed on 
the entrance screen of the image intensifier. Thus, collection 40 is formed of 
consecutive images 41 \ to 41 n which mutually overlap to various degrees. The overlap 
between sub-images depends on the displacement between positions of the x-ray 
source at the irradiation for forming said sub-images. 



Please replace the paragraph on page 6, lines 19-28, with the following amended 
paragraph: 

An apparatus configured according to the present invention accurately merges 
the sub-images of collection 40 into assembled image 42 44, which contains a shadow- 
image of the patient's entire legs. For example, images 41 n -i and 41 n need to be 
brought into spatial registration before merging, mis-registration being due, for example, 
to planned or accidental changes in the orientation of x-ray beam 15 when the two 
projection images are formed. Mis-registration is reflected in area 42 of overlap where, 
for example, point 43 n -i is at a different location than point 43 n , although both points 
represent the same feature in patient leg 16. Mis-registration is corrected by 
determining a projective transformation the that relates images 41 n -i and 41 n so that 
points 43 n -i and 43 n are at the same spatial position. Images in spatial registration can 
be merged without blurring. 



Please replace the paragraph on page 7, lines 22-31, with the following amended 
parag raph: 



Turning now to a more detailed description of the individual steps of the 
methods, selection of feature points in the overlapping region of a pair of overlapping 
2D images at step 52 can, for example, be done manually. Here, a user, for example, 
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at apparatus 29, selects N easily distinguishable points in the 3D scene that appear in 
both images. Then these feature points are identified in both images, and their 
coordinates are measured in both the images. This results in N pairs of 2D 
v Jh coordinates, each pair being the corresponding coordinates of a feature point. These 



are represented by the pairs: 

U, = {u\j U2,J ^ Yi = (vi,,- V 2 ,if i = 1 , . . . , N (1) 

Here, the Uj are the coordinates of points in one image, and the Vj are the 
coordinates of corresponding points in the other image. 



\0 



Please replace the paragraph beginning on page 7, Iine32 through page 8, line 5, 
with the following amended paragraph: , 



Alternatively, the feature points may be selected automatically by the image 
processing apparatus. One automatic method proceeds by first sparsely sampling 
points in the overlapping region of the image, then using matching or correlation of 
locally surrounding blocks of points to determine candidate pairs of sampled points that 
should correspond i ng correspond to the same 3D scene point, and finally retaining only 
those candidates pairs that have sufficient surrounding image structure for accurate 
block matching or correlation. See, e.g., Schultz et al., 1999, IEEE International 
Conference on Acoustics. Speech and Signal Processing , vol. 4, pp. 3265-3268. 

Please replace the paragraph on page 8, lines 6-13, with the following amended 

paragraph: 

Another automatic method proceeds by first constructing multiresolution 
decomposition of the images by self-similar discrete wavelet transforms, then selecting 
candidate image points having local maximums of the pixel-value gradient greater than 
a threshold where the pixel-value gradient is prominent at all resolutions, and finally 
retaining only those pairs of candidates points that have sufficiently cross-correlated 
locally surrounding blocks of points, and thereby that should correspond to the same 
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^£^L l -*l feature in the 3D scene. The conditions imposed on the pixel-value gradient are to 

insure that there is sufficient image structure surrounding the candidate points for 
k accurate cross-correlation = — — ■ 



KG 



Please replace the paragraph on page 8, lines 14-25, with the following amended 
.paragraphs — — — — — 



The next steps determine a single substantially optimal projective transformation 
that relates the pairs of corresponding points in the overlapping region of two images. 
A projective transformation best models the relation between the pairs of overlapping 
images because, as detailed above, the images are obtained by devices that project a 
3D scene onto the 2D images. A projective transformation linking pairs of 
corresponding points is represented i n the fo l lowing by the following matrix equation. 

A-u, + T 

V/ = — 7 - 

C^iii + l (2) 

This transformation has the following matrix parameters (A. C, T) which are determined 
by their eight matrix elements. 



^#21 Ct22j 



f C = (ci c 2 y j T = (u t 2 J (3) 

This transformation is selected to hold for all N pairs of corresponding 2D coordinates 
- previously se lected 

Please replace the paragraph on page 9, lines 11-17, with the following amended 
paragraph: 

It is an important discovery on which this invention in founded that the stability 
1 P^^l^ and accuracy of the solution for the parameter matrix, P, is considerably improved if the 
coordinate translations are chosen in order to minimize, on average, the numerical 
range of all the coordinates values of the corresponding feature points in each image. 
Such minimization is effective because it reduces computational errors in the terms 
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involving the products of feature point coordinates appearing in the matrices Ui, these 
product te rms being apparent in £qnsr Egyations 4A, 4b and 5. 

Please replace the paragraph on page 10, lines 6-12, with the following amended 
paragraph: 

In an alternative embodiment, this translation can be avoided in the original 
coordinate systems m if the two images are chosen so that the numerical ranges of the 
feature point coordinates are initially minimized. This choice can be done by searching 
for optimum placement of the origins of the coordinate systems in the two images. This 
search for the coordinate origin can seek to minimize any of Bm&r Equations 9a, 9b or 
9c, or another metric such as the mean square distance of the feature points from the 
coordinate origin. If the coordinate systems are so chosen, the projective 
transformation can be directly determined. 

Please replace the paragraph on page 12, lines 5-11, with the following amended 
paragra ph: 

The methods of this invention are readily implemented on image processing 
apparatus, for example, apparatus 29, by programming the above steps in an 
appropriate programming language, for example, C or FORTRAN. Initialization and 
termination activities of such programs occur in steps 50 and 58, respectively. 
Optionally, numerical algebra packages, such as LINPACK, can be used to rout i n e 
routinely perform various ones of the above steps, such as necessary matrix 
multiplication, finding the SVD and so forth. One of skill in the art can readily and 
ro.utinely.perfQ^^ of the above des cription. 

Please replace the paragraph on page 13, line 1, with the following amended 
paragraph: 



TEST FEATURE PONT POINT COORDINATES 
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